Repository navigation
FINAL clause speed: refresh for ClickHouse 23.2-26.10 - #194
Merged
Merged
Conversation
… and lazy FINAL notes Fix wrong versions (23.3, 24.5/24.6 vertical FINAL, 25.5/25.6 skip indexes, 25.11, row policy defaults), add missing changelog entries, a skip-index exact-mode note (PR 121799) and a lazy FINAL vs lazy materialization section. Benchmarks and methodology are in the PR description. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1KN5umrvcVcdzrL77H4k3
Contributor
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FINAL clause speed article: changelog refresh + benchmarks
Updates
content/en/altinity-kb-queries-and-syntax/altinity-kb-final-clause-speed.mdfrom the ClickHouse changelogs (23.2 to 26.10), with every setting default and version checked againstsystem.settings/system.settings_changesin official Docker images (25.8, 25.12, 26.1 to 26.10).Corrections to the old article
enable_vertical_finalwas off in 24.5 and on again from 24.6;use_skip_indexes_if_final_exact_modeadded 25.5 (default 25.6); skip-index PK-intersection check removal is 25.11 (not 26.1);apply_row_policy_after_finaldefault 1 in 25.12, 0 in 26.1, 1 from 26.2.analyzer_compatibility_apply_final_to_all_joined_tables=1restores); reverse-order FINAL read from 26.9;defer_partition_pruning_after_finalchanges results as described;final=1, PREWHERE-before-FINAL and row policy behaviour as described.Benchmark 1: PR #121799 (skip index, exact mode), 26.8.7.19 vs 26.10.3.7
Host gpu-01 (20 cores, 62 GB), container
--memory 12g --cpus 8, one version at a time, 8M rows, 1 warmup + 5 runs, mark/uncompressed caches dropped before each run (OS page cache warm). QuerySELECT count(), sum(key) FROM t FINAL WHERE v = 1. Results equaluse_skip_indexes=0in all 18 configs; selected marks and read_rows identical on both versions. Medians, ms:Other: index type (minmax 417/103, set 429/117, bloom_filter 425/114) does not matter; non-overlapping parts still gain (granularity 8: 176 -> 25; granularity 64: 28 -> 8); index column in the PRIMARY KEY: 24 vs 38 ms, both skip the recovery pass; merged parts + 20 fresh parts at granularity 64, 1/2000: 166 vs 153 ms; composite ORDER BY (a,b), 20 parts, granularity 64: 80 -> 24 ms. Baselines on 26.10 (PR layout): exact 103,
use_skip_indexes_if_final_exact_mode=0255,use_skip_indexes_if_final=01064; at 1/200 exact (1547) is slower than no index (1101).Not explained: exact mode faster than approximate mode on 26.10, and the non-overlapping gain. My PR-layout run selected 32057 marks and 417/103 ms, not the PR's 65466 and 3.88 s/0.76 s. Recovery-pass time alone was not captured.
Repro (PR perf test):
Benchmark 2: lazy materialization / lazy FINAL (PRs #101647, #110722, #110576), 26.4.5.143, 26.7.38.18, 26.8.36.3, 26.10.3.7
Host llmbox-01 (24 cores, 214 GB), container
--memory 16g --cpus 8,max_threads=8, 1 warmup + 5 runs, medians, warm caches (no root: OS page cache not dropped).t_w:ReplacingMergeTree(ver) ORDER BY k, 4 unmerged parts x 2M keys, 3 x String(200). Combos: N = everything off, D = defaults (lazy materialization on), L = lazy FINAL on, LN = lazy FINAL on, lazy materialization off. Median ms N/D/L/LN:SELECT *WHERE f=7 LIMIT 10SELECT s1WHERE f=7 LIMIT 10k,f, LIMIT 10id=, LIMIT 100SELECT *, LIMIT 10Read bytes on 26.8,
SELECT *LIMIT 10: 899 MB (N) -> 98 MB (D). Lazy FINAL helped only for a contiguous key range (c=5, no LIMIT: 428 -> 24 ms on 26.7, 5143 -> 95 MB) and a unique id without LIMIT (432 -> 26 ms); no-LIMIT cost at 10% selectivity 175 -> 245-260 ms and at 50% 487 -> 635-665 ms; worst case 26.7f<1000 LIMIT 118 -> 93 ms. Plan lines:JoinLazyColumnsStepoverLimit -> Filter -> ReadFromMergeTree (FINAL: 1)(not present on 26.4, nor without a filter, nor with LIMIT abovequery_plan_max_limit_for_lazy_materialization= 10000; verified locally). Correctness: no wrong results or exceptions on any version for Nullable PK with NULLs,is_deletedwith PREWHERE,hasTokentext index (plan not captured).Not tested / dropped
make_distributed_plan(private preview, needs stateless workers); 25.1/25.10 FINAL performance entries are changelog-only.Tested only against official Docker images; nothing was built with Hugo locally (CI-pinned 0.128.2).
🤖 Generated with Claude Code
https://claude.ai/code/session_01D1KN5umrvcVcdzrL77H4k3